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The connectivity of complex networks and functional implications has been attracting much interest in many 
physical, biological and social systems. However, the significance of the weight distributions of network links 
remains largely unknown except for uniformly- or Gaussian-weighted links. Here, we show analytically and 
numerically, that recurrent neural networks can robustly generate internal noise optimal for spike 
transmission between neurons with the help of a long-tailed distribution in the weights of recurrent 
connections. The structure of spontaneous activity in such networks involves weak-dense connections that 
redistribute excitatory activity over the network as noise sources to optimally enhance the responses of 
individual neurons to input at sparse-strong connections, thus opening multiple signal transmission 
pathways. Electrophysiological experiments confirm the importance of a highly broad connectivity spectrum 
supported by the model. Our results identify a simple network mechanism for internal noise generation by 
highly inhomogeneous connection strengths supporting both stability and optimal communication. 

The dynamics of a complex network depend crucially on the particular connection architecture of the 
network 1 " 5 . In the absence of sensory stimulation, cortical networks are far from silent, but generate rich 
and ubiquitous forms of electrical activity that represent noisy internal brain states. Such states typically 
display low-frequency (<10 Hz, typically 1-2 Hz) irregular neuronal firing 6 " 9 , interact bidirectionally with 
sensory experience 10 " 15 , and, moreover, involve a rich repertoire of complex sequential activity patterns 1617 . 
There has been much recent interest in the genesis 18 " 24 and function 10 " 13 of spontaneous activity or noise in 
the brain, since noise may be the basic mechanism underlying our percept and decision process, which are 
essentially probabilistic 14,15 . While the role of network connectivity in complex neural dynamics has been studied 
extensively 4,25 " 28 , weighted networks have been investigated only recently 29-32 and the dynamical and functional 
implications of the distribution of link weights remain largely unknown in excitable systems. 

Recent experiments revealed that the amplitude of excitatory synaptic potentials (EPSPs) between cortical 
pyramidal neurons obeys a long- tailed, typically lognormal, distribution 33,34 . Such a distribution creates a synaptic 
spectrum spanning from vast numbers of weak synapses (typically, the amplitude of EPSP < 1 mV) to a small 
fraction of extremely strong synapses, for which EPSP amplitude can be several millivolts. Here, we numerically 
and analytically study the significance of these strong-spare and weak-dense (SSWD) connections for the 
dynamics of recurrent networks, in which the weights of recurrent excitatory synaptic inputs to each neuron 
obey a long-tailed distribution. We asked whether reverberating synaptic input generated by such a distribution is 
sufficient for the genesis of stable spontaneous activity, and whether this internal noise provides an optimal 
solution for efficient information processing. 

Results 

The dynamics of each neuron are described by a leaky integrate-and-fire model: 

dv 1 



-(v-V L )-g E (v-V E )- gl (v-V L ) 



(1) 



where v is the membrane potential. The membrane time constant T m is 20 [ms] for excitatory neurons and 10 [ms] 
for inhibitory neurons, and the reversal potentials of leak, excitatory and inhibitory postsynaptic currents are 
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Figure 1 | Maximizing the fidelity of spike transmission with long-tailed 
sparse connectivity, (a) Each excitatory neuron has a lognormal amplitude 
distribution of EPSPs. The resultant mean and variance of the model are 
0.89 [mV] and l.l 2 [mV 2 ], respectively, whereas those shown in a previous 
experiment [1] were 0.77 [mV] and 0.9 2 [mV 2 ]. Inset is a normal plot of the 
same distribution, (b) Schematic illustration of the neuron model with 
strong-sparse and weak-dense synaptic inputs. Colors (red, green and blue) 
indicate inputs to the top three strongest weights, (c) C.C.s between the 
output spike train and input spike trains at the 1st (red), 2nd (green) and 
3rd (blue) strongest synapses on a neuron are plotted against the mean 
membrane potential and the corresponding input firing rate at each synapse. 
The dashed line and shaded area show the mean and SD of the membrane 
potential distribution of excitatory neurons shown in Fig. 2f for the SS WD 
network. Vertical bars represent SEM over different realizations of random 
input. The dashed line indicates an analytical curve for the strongest synapse 
of the long-tailed distribution, while the dot-dashed line is the C.C.s for the 
strongest synapse when EPSP amplitudes obey Gaussian distribution, (d) 
Similar C.C.s obtained by dynamic clamp recordings from a cortical neuron. 
The color code and vertical bars are the same as in C. (e) The trial- averaged 
C.C.s for the strongest synapses on n=14 neurons. 

V L = -70 [mV], V E =0 [mV], V T = -80 [mV], respectively. The excit- 
atory and inhibitory synaptic conductances^ and^ [ms -1 ] normal- 
ized by the membrane capacitance obey 



dt 



+ E G ^E^- 5 ;-4-)> X = E,I (2) 



where S(t) is the delta function, Gp dp Sj are the weight, delay and 
spike timing of synaptic input from the j-th neuron, respectively. 
The decay constant t s is 2 [ms] and synaptic delays are chosen 
randomly between d 0 — 1 to d 0 +l [ms], where d 0 = 2 for excit- 
atory-to-excitatory connections and d 0 = 1 for other connection 
types. The values are determined from the stability of spontaneous 



activity (Methods). Spike threshold is Vth r = —50 [mV] and v is reset 
to V r = —60 mV after spiking. The refractory period is 1 [ms]. 

The values of Gi for excitatory-to-excitatory connections are dis- 
tributed such that the amplitude of EPSPs x measured from the 
resting potential obey a lognormal distribution 



p{x) = 



exp[-(logx-^) 2 /2ff 2 ] 



2nax 



(3) 



on each neuron (Fig. la), where the values o= 1.0 and fi-a 2 = log(0.2) 
well replicate the experimentally observed long- tailed distributions 
of EPSP amplitudes 33 ' 34 . We declined any unrealistic value of G z that 
gives an amplitude larger than 20 [mV] by drawing a new value 
from the distribution. The resultant amplitude of strongest EPSP 
was about 10 [mV] on each neuron. For simplicity, excitatory- 
to-inhibitory, inhibitory-to-excitatory and inhibitory-to-inhibitory 
synapses have uniform values of G,-= 0.018, 0.002 and 0.0025, 
respectively. Excitatory-to-excitatory synaptic transmissions fail at 
an EPSP amplitude-dependent rate of p E = a/(a+ EPSP), where 
a=0.1 [mV] 34 . 

We first demonstrate numerically that the long-tailed distribution 
of EPSP amplitudes achieves aperiodic stochastic resonance for spike 
sequence on a single neuron receiving random synaptic inputs 
(Fig. lb). Stochastic resonance refers to a phenomenon wherein a 
specific level of noise enhances the response of a nonlinear system to 
a weak periodic or aperiodic stimulus 35 " 37 , and has been observed in 
many physical and biological systems 38 " 45 . We vary the average mem- 
brane potential of the neuron by changing the rate of presynaptic 
spikes at a portion of the weakest excitatory synapses (EPSP ampli- 
tudes < 3 mV). Interestingly, the cross -correlation coefficients 
(C.C.) between output spikes and inputs to the strongest synapses 
are maximized at a subthreshold membrane potential value about 10 
[mV] above the resting potential and 10 [mV] below firing threshold 
(Fig. lc). At more hyperpolarized levels of the average membrane 
potential, even an extremely strong EPSP (—10 mV) cannot evoke a 
postsynaptic spike, and the fidelity of spike transmission is reduced. 
On the contrary at more depolarized average membrane potentials, 
the neuron can fire without strong inputs, also degrading the fidelity. 

We can express the C.C.s in terms of the conditional probability of 
spiking by strong- sparse input, which we can analytically obtain 
from the stochastic differential equations for weak-dense synapses 
(Methods). The analytic results well explain the optimal neuronal 
response obtained numerically (Fig. lc). The phenomena can be 
regarded as stochastic resonance for aperiodic spike inputs 36 ' 37 . We 
find that the stochastic enhancement of spike transmission is much 
weaker in a neuron (Fig. lc, dashed curve) having Gaussian- distrib- 
uted EPSP amplitude, which give the same mean and variance of 
synaptic conductances as the lognormal distribution but no tails of 
strong synapses (Supplementary Methods). The results prove the 
advantage of long-tailed distributions of EPSP amplitude. 

We confirmed the above model's prediction by performing 
dynamic clamp recordings from cortical neurons (n= 14). To mimic 
synaptic bombardment with long-tailed distributed EPSP ampli- 
tudes, we injected the synaptic current given in equation (2) by using 
the same values of excitatory and inhibitory conductances as used in 
Fig. lc (Supplementary Methods). The rate of random synaptic 
inputs was varied in a low-frequency regime. The physiological result 
also demonstrated the maximization of the fidelity of synaptic trans- 
mission (Fig. Id, e). 

Now, we ask whether the above stochastic resonance is achievable 
by the noise generated internally by SSWD recurrent neural net- 
works. To see this, we conduct numerical simulations of equations 
(1) and (2) for a network model of 10000 excitatory and 2000 
inhibitory neurons that are randomly connected with coupling 
probabilities of excitatory and inhibitory connections being 0.1 and 
0.5, respectively. Since the network has a trivial stable state in which 
all neurons are in the resting potentials, we briefly apply external 
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Figure 2 | Spontaneous noise in the SSWD recurrent network. The 

network receives neither external input nor background noise, and hence 
activity is spontaneous, (a) Upper, Spike raster of excitatory (red) and 
inhibitory (blue) neurons in the noisy spontaneous firing state. Lower, The 
population firing rates of excitatory (red) and inhibitory (blue) neurons. 

(b) Firing rate distributions of excitatory (red) and inhibitory (blue) 
neurons can be fitted by lognormal distributions (black lines). Mean firing 
rates are 1.6 and 14 [Hz] for excitatory and inhibitory neurons respectively. 

(c) CVs of inter-spike intervals are distributed around unity in excitatory 
(red) and inhibitory (blue) neurons, (d) Time courses of the membrane 
potentials of excitatory (red) and inhibitory (blue) neurons exhibit large 
amplitude fluctuations, (e) Scatter plot of the instantaneous population 
activities of excitatory and inhibitory neurons. The solid line represents 
linear regression, (f) Distribution functions of the fluctuating membrane 
potentials show the depolarized states of excitatory (red) and inhibitory 
(blue) neurons, (g) The mean (solid) and standard deviation (dashed) of 
the membrane potential fluctuations of an excitatory neuron when all 
EPSPs smaller than the minimum value given in the abscissa are 
eliminated. Here, we remove a portion of excitatory synapses on a neuron 
from the weakest ones. 



Poisson spike trains to all neurons during initial 100 [ms] to trigger a 
spontaneous firing. In the absence of external input, the model sus- 
tains a stable asynchronous firing initiated by a brief external stimu- 
lus (Fig. 2a). The spontaneous network activity emerges purely from 
reverberating synaptic input, is stable in a very low-frequency regime 
(Fig. 2b) and is highly irregular (Fig. 2c) as experimentally 
observed 6 ' 8 ' 9 . Firing rate distributions are well fitted by lognormal 
distributions 7 ' 46 ' 47 . Each neuron exhibits large membrane potential 
fluctuations, on top of which spikes are generated occasionally 
(Fig. 2d), owing to the dynamic balance between excitatory and 
inhibitory activities (Fig. 2a and 2e) 18,20 ' 24,48 . All these properties are 
consistent with the spontaneous activity observed in cortical neu- 
rons 20 . Importantly, the average values of the membrane potentials 



are around -60 mV in excitatory neurons (Fig. 2f) 20 ' 49 , at which spike 
transmission at strong- sparse synapses becomes most reliable 
(Fig. la, shaded area). Inputs to weak- dense synapses maintain the 
average membrane potential of each neuron (Fig. 2g), whereas inputs 
to strong- sparse synapses govern sparse spiking. Therefore, weak- 
dense and strong- sparse synapses have different roles in stochastic 
neural dynamics, although they distribute continuously. 

Long-tailed distributions of coupling strengths offer a much wider 
region of the parameter space to stable spontaneous activity than 
Gaussian -distributed coupling strengths (Supplementary Fig. 1). 
Furthermore, a linear stability analysis reveals the homeostasis of 
the ongoing state of the SSWD network (Methods). 

What is the underlying mechanism and functional implications 
of the spontaneous noise generation? Strong-sparse synapses form 
multiple synaptic pathways in the recurrent neural network 
(Fig. 3a). Owing to the stochastic resonance effect at these 
synapses, spike sequences are routed reliably along these pathways 
(Fig. 3b: Supplementary Methods) that may branch and converge 
(Fig. 3c). Since strong synapses are rare, spike propagation along a 
pathway is essentially unidirectional, as indicated by the cross- 
correlograms for presynaptic and postsynaptic neuron pairs 
(Fig. 3d). If, therefore, external stimuli elicit spikes from the initial 
neurons of some strong pathways, the spikes can stably travel 
along these pathways without much interference (Fig. 3e). The 
number of spikes received at the end of a pathway is proportional 
to that of spikes evoked at the start, although fluctuations in the 
spike number increase with the distance of travel (Fig. 3f). These 
results imply that spikes can carry rate information along the 
multiple synaptic pathways embedded by strong-sparse synapses. 
The presence of precise spike sequences has been reported in the 
brain of behaving animals 50 " 52 . We note that the same spikes are 
sensed as noise if they are input to weak synapses. 

Discussion 

In this study, we have explored a coordinating principle in neural 
circuit function based on a long- tailed distribution of connection 
weights in a model neural network. The network properties con- 
ferred by the long- tailed EPSP distribution account for a role of noise 
in information routing and present a novel hypothesis for neural 
network information processing. Namely, we have demonstrated 
that a single neuron shows spike-based aperiodic stochastic res- 
onance; the cross -correlation coefficient between output spikes of a 
single neuron and inputs to the strongest synapses are maximized 
when the neuron receives a certain amount of background noise. 
Stochastic resonance has been studied in neuronal systems in various 
contexts. The presence of sensory noise improved behavioral per- 
formance in humans 38 ' 41 and other animals 39 . Synaptic bombard- 
ment enhanced the responsiveness of neurons to periodic sub- 
threshold stimuli 20 ' 40 ' 42 . Asynchronous neurotransmitter release can 
give a noise source for stochastic resonance in local circuits of model 
neurons with short-term synaptic plasticity 43,44 . A surprising result 
here is that the networks may internally generate optimal noise with- 
out external noise sources for the spike-based stochastic resonance 
on sparse-strong connections. Weak-dense connections redistribute 
excitatory activity routed reliably on strong connections over the 
network as optimal noise sources to sustain spontaneous firing of 
recurrent networks. 

Internal noise or asynchronous irregular firing may provide the 
neural substrate for probabilistic computations by the brain, and how 
such activity emerges in cortical circuits has been a fundamental 
problem in cortical neurobiology. Such neuronal firing has been 
replicated by sparsely connected networks of binary or spiking 
neurons 1819,21 " 23 , and the importance of excitation-inhibition balance 
has been repeatedly emphasized. However, the mechanism to 
generate extremely low- rate spontaneous asynchronous firing 
(<C10 Hz) remained unclear, and our model gives a possible solution 
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Figure 3 | Spike information routing in the SSWD recurrent network model, (a) A schematic illustration of the SSWD recurrent network. 
Thick lines stand for strong-sparse connections and thinner lines for weak-dense connections. In reality, the strength of connections is 
continuous obeying a long-tailed distribution, (b) Examples of spike sequences routed in the network are shown by different colors. Insets 
magnify the raster plot, (c) Examples of branching (left) and converging (right) pathways formed by the strong synapses. Numbers refer to 
neurons, and colors to the corresponding pathways in (b). (d) Cross-correlograms are averaged over strongly connected neuron pairs (EPSP 
>8mV). (e) Repeated external stimuli (arrows) evoke simultaneous spike propagations in two pathways, (f) Linear relationship between the 
number of input spikes and that of output ones in a pathway. The dashed line and vertical bars represent linear regression and SD over trials, 
respectively. 



to this. A large-scale model of mammalian thalamocortical systems 
consisting of a million neurons with realistic electrophysiological and 
morphological properties generated asynchronous irregular states 23 , 
implying that interactions between dynamical and anatomical pro- 
cesses significantly contribute to internal noise generation. By con- 
trast, such states appears in our model from a special synaptic 
connectivity within local cortical circuits. It is worth while noting 
that the asynchronous irregular firing of our model does not rely on 
slow synapses like NMD A receptor- mediated ones. Though slow 
synapses may improve the stability of such states, our model suggests 
that such a role of slow synapses is subsidiary. 

Long- tailed amplitude distributions of EPSPs can arise from activ- 
ity-dependent synaptic plasticity. In networks of rate neurons with 
linear response functions, a Hebbian learning rule induces a lognor- 
mal weight distribution when the rule of weight increment depends 
nonlinearly on the weights 53 . In networks of spiking neurons, spike- 
timing dependent plasticity results in a long-tailed conductance dis- 
tribution if the weight dependence for long-term depression 54 depends 
sublinearly on synaptic weights 55 . Activity- dependent plasticity may 
switch and reroute different pathways of strong synapses due to 



sensory or motor experiences of animals while total distribution of 
EPSP amplitude of the network are kept intact. It is intriguing whether 
the activity-dependent pathway rerouting may provide a mechanism 
to represent Bayesian priors of sensory experiences in spontaneous 
cortical activity 15 and habitual motor coordination 56 . 

In summary, we conjoin two fundamental principles in signal pro- 
cessing and complex phenomena observed in cortical neural networks: 
stochastic resonance and noisy internal brain states. The key of this 
link is the coexistence of a spectrum of strong-sparse and weak-dense 
connections that gives a mechanism by which excitable networks 
generate and maintain optimal noise level for efficient spike commun- 
ication. These results have implications for a role of noise in networks 
with a broad spectrum of coupling strengths, such as the gating of 
specific signal pathways with the probabilities of pathway selection 
modulated by the dynamics of internal noise generation. 

Methods 

Cross-correlation coefficient. We can analytically calculate cross-correlation coefficients 
by assuming that spike trains are well approximated by a low-rate Poisson process. Then, 
the cross-correlation coefficient between input and output spike sequences is estimated as 
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Pr(xi n x out ) T — Tin T ou t T 2 



(4) 



where r in and r out are firing rate of input and output sequences respectively, T is a short 
period of time satisfying r^T-Cl and r 0lrf T<O, and Pr(x out \xi n ) is the conditional 
probability of output spike for given input spike at strong synapses. In numerical 
simulation, we evaluated Pr(x out \x in ) by detecting a post-synaptic spike within the epoch 
of EPSP rise from the arrival of an input spike. 

Analytical solution of the cross-correlation coefficient. We can analytically 
calculate the firing rate and cross-correlation coefficient of each neuron by dividing 
excitatory synaptic inputs to the neuron into two parts, one consisting of weak and 
modestly strong synapses and one consisting of extremely strong synapses. In this 
approximation, we may treat inputs to the former excitatory synapses and inhibitory 
synapses by the diffusion approximation 57 , in which Poisson spike inputs on (2) are 
replaced with a white Gaussian noise having the same mean and variance of the 
Poisson inputs. We then used the effective-time-constant approximation 58 to replace 
v— V E and v— Vj with y 0 — V E and V 0 — V I to obtain linear stochastic differential 
equations 

' dv (v-Vb) , 
- = Vr\ 

d J = - n - + oi 
at t s 



where V 0 = (V L /T m + V E g E0 + Vjg m )x e and T e =l/(l/x m +gEo+gro) are me equilibrium 
membrane potential and the effective membrane time constant, respectively. The 
mean excitatory conductance of the first group and mean inhibitory conductance are 
gEo = Yl Q ( 1 — pi ) Y{ and g I0 = xMfii^b respectively, in terms of the firing rate 

z'elst group 

of the i-th excitatory synaptic input r b the synaptic transmission failure p b firing rate 
at inhibitory synapses 77 and the number of inhibitory synapses on the neuron M h The 
fluctuation of the total synaptic current rj is given as 
a 2 = (V Q - V E f Y, G l( l ~Pi) r i + ( y o - V/) 2 MjGf £/. The stationary 

z'elst group 

probability density of the membrane potential and the output rate are obtained from 
equation (5) as 59 



P(v) = C exp 



^ 2 'U • 



V<Vreset 
V reset <V <V thr 



■ref 



e Jn 1 2 / (1 + erf (s))ds 



(6) 



(7) 



up to the first order of \/T s /x my where erf(x) is the error function and 

Sl = (V reset ~ V^/C^a/V 7 ) + « V ?s/?e / 2, S 2 = (V thr - Vq)/(t s y/%~ e o) + a yj X s /x e 1 2, 

s(v) = (v — Vo)/ (^sv^e 0 ")' a = v / 2|^(l/2)| and E, is the Riemann zeta function. 
Normalization constant C is calculated from the P(v)dv=l—x r ef^out,m- 

The contribution of extremely strong excitatory synapses to output firing is 
approximated as the sum of the conditional firing probabilities over inputs to these 
synapses. In the effective-time-constant approximation, the effective amplitude of 
EPSP evoked by z'-th synapse is 



Ee,M- 



(Vb-v)t; 



,W(T £ -T S ) -T S /(T e -T S ) 



(V E 



-V l )t 



W(t,„ 



-T s /(T m -T s 



Ei = 



(Ve-v) 
(Ve-V l ) 



PEi 



(8) 



where v is the membrane potential just before the arrival of presynaptic input and E t is 
the EPSP amplitude measured from the resting potential. Because the conditional pro- 
bability of having an output spike given the z'-th input is equal to the area of the stationary 
density function satisfying v + E e j(v) > Vth r > the conditional probability is equal to 

J { v E- v L) v thr-P E i V E 
(V E -V L )-PEi 

Then, by summing these contributions, we obtain the firing rate of the neuron as 



= r 0 ut, w + Yl Pirr= 

ze2nd group 



t,w ?oi 



(10) 



Finally, by substituting P t of the strongest synapse into P(x out \x in ) and using equation 
(10), we obtain an analytical expression of the correlation coefficient given in equation 
(4). To derive the analytical curve shown in Fig. lc, we classified the five strongest 
synapses into the second group and the remaining ones into the first group. 



Stability of sparse spontaneous activity. Using the above analytic results, we can 
derive a coupled evolution equation for the average firing rates of excitatory and 
inhibitory neuron pools. Since neurons are connected in a non-biased manner, we can 
use the mean- field approximation. 

The firing rates of excitatory and inhibitory neuron pools, r E and 77 can vary in time 
due to interactions between them. Since the mean output rate is equal to the mean rate 
of input given synaptic delays in a recurrent network, we may represent the time 
evolution of the mean firing rates of these neuron pools as the following relaxation 
process with their effective membrane time constants: 

' dv^ 1 

7 = (rE(t)-r ouUE (r E (t-d EE ),ri( t -dEi))) 

I n r . (") 



dt 



T e ,I 



-{ri(t)-r out ,i{ri(t-d m ),ri{t-du))) 



where r outjE (r E ,r I ) = r outjS +r out)W and r ouU (r E ,r T ) is similar to r out)W if we replace the 
average excitatory-to-inhibitory and inhibitory-to-inhibitory synaptic conductances 
with their unique values in equation (7). Constants d X Y represent the mean synaptic 
delays from Y neurons to X neurons (X, Y = E or I). We can obtain the nullclines of r E 
and rj shown in Supplementary Fig. 2 by equating the left-hand sides of equation (11) 
to zero. Sustained spontaneous activity may correspond to a nonzero intersection 
(teo> fro) of me tw0 nullclines. The stability of spontaneous activity can be studied by 
the linear stability analysis: we substitute r E (t)=r E0 +se Ax and ri(t)=r w + Ece A in 
equation (11) and expand the equation up to the linear order in e. Then, the elim- 
ination of s and c from the resultant equation gives a closed-form equation for the 
stability index I: 60 



(^Te, E + l-a EE e- 2TEE )(^T e ,l + l- 



(12) 



We solve Equation (12) numerically to obtain the stable region in the parameter space 
(Supplementary Fig. 2). The results of the theoretical analysis well coincide with those 
of numerical simulations. 
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